Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporary decorator to mark & log 'daskified' methods #245

Merged

Conversation

sadielbartholomew
Copy link
Member

One final PR for some setup and helper code related to the LAMA to Dask migration (#182) before I start pushing up the actual method migrations in turn!

This one adds a temporary custom decorator that will be applied to directly mark methods that we consider to have been migrated fully in the code, so we don't have to cross-reference with the very long checklist in the issue (though I will make sure to update the list too upon each corresponding merged PR). As well as acting as a marker, the decorator provides a log message reported each time the method is run (which can be adapted to provide more info, as useful) and allows the log level to be tuned for those methods only by simply changing the decorator argument value.

I have marked all methods considered 'daskified' after @davidhassell's initial work with the decorator and will add it incrementally to other methods as they are completed.

Example of resulting test output change

By running pytest test_Data.py -rP (the options of which I have found useful so far), STDOUT is captured, e.g. under the configuration of this branch state it returns:

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/sadie/cf-python
collected 73 items                                                             

test_Data.py ssss.s..ss.s.s.ssssssssssssssss.sssssssssssssssssssssssssss [ 80%]
ssssssssssssss                                                           [100%]

==================================== PASSES ====================================
________________________ DataTest.test_Data___setitem__ ________________________
------------------------------ Captured log call -------------------------------
WARNING  cf.data.data:data.py:155 %%%%% Running daskified hardmask %%%%%
WARNING  cf.data.data:data.py:155 %%%%% Running daskified __setitem__ %%%%%
WARNING  cf.data.data:data.py:155 %%%%% Running daskified shape %%%%%
...
...
WARNING  cf.data.data:data.py:155 %%%%% Running daskified array %%%%%
WARNING  cf.data.data:data.py:155 %%%%% Running daskified hardmask %%%%%
WARNING  cf.data.data:data.py:155 %%%%% Running daskified size %%%%%
...
...
======================== 7 passed, 66 skipped in 0.98s =========================

though the minimal test output report of pytest can be recovered by removing the -rP options, and/or converting the arguments to daskified() to sto the message output.

Overall I'm setting us up for some major debugging sessions that will probably be required in later stages of the migration 😄

@sadielbartholomew sadielbartholomew self-assigned this Aug 19, 2021
@sadielbartholomew sadielbartholomew changed the title Temporary decorator to mark and log 'daskified' methods Temporary decorator to mark & log 'daskified' methods Aug 19, 2021
@sadielbartholomew
Copy link
Member Author

CI jobs are all happy under the new workflow for this branch i.e. the migration. No review required as it is setup code.

@sadielbartholomew sadielbartholomew merged commit 68e8a73 into NCAS-CMS:lama-to-dask Aug 19, 2021
@sadielbartholomew sadielbartholomew deleted the lama-to-dask-2 branch August 19, 2021 20:21
@davidhassell davidhassell added the dask Relating to the use of Dask label Oct 5, 2021
@davidhassell davidhassell added this to the 3.14.0 milestone Nov 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dask Relating to the use of Dask
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants